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M4D 


REPORT ON THE MOBILES FOR 
DEVELOPMENT (M4D) COURSE 


For six weeks beginning October 2, 2013, IIT Kanpur and Commonwealth of Learning 
delivered a massive open online course (MOOCJ on mobiles for development (M4D). This 
report provides a pedagogical review of the M4D online course. A review of the M4D 
course design and delivery experience will help COL staff to better understand the 
prototype course, its participants, their performance and their response to the course 
design. 
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Executive summary 

For six weeks beginning October 2, 2013, IIT Kanpur and Commonwealth of Learning (COL) delivered a 
massive open online course (MOOC) on the topic of mobiles for development (M4D). This report provides a 
pedagogical review of the M4D online course to assist IIT and COL staff to better understand the outcomes 
of the prototype course, the participants, their participation and performance, and their response to the 
course design. 

KEY FINDINGS 

Relevance 

The IIT and Col M4D team demonstrated that successfully offering large online courses did not have to use 
custom and "branded” MOOC platforms of the type used by well-known MOOC providers such as Coursera 
and Udacity. The open source software platform Sakai was used successfully to provide a MOOC-style 
course to 1,441 course participants. 

Successes and impacts 

The course attracted 2,282 registrants from 116 countries. About 63 percent of the registrants were 
considered to be "active” in the course space (1,441 students). 

A total of 333 students were eligible to receive certificates of Competence (244 students) or Participation 
(89 students). Certificates (P or C) were awarded to approximately 23 percent of active participants in the 
M4D course. 

The top five countries in terms of registrants were India, Nepal, Mauritius, Grenada and South Africa. About 
500 registrants were from countries in the Africa, Caribbean, and Pacific regions. About 200 registrants 
were from the OECD countries, and from Eastern Europe. 

Design and delivery 

The team demonstrated that teaching quality and content were critical factors in keeping participants 
interested. The team also demonstrated that high availability of online services is a critical factor to keep 
students as participants in a MOOC. 

Online mentoring /facilitation was a key investment of time for which instructors should be trained and 
prepared. ODL institutions may have a significant starting advantage in offering MOOCs because of their 
inherent strengths with pedagogy, resources and expertise to mount large-scale learning programs. 

Cost-effectiveness 

The M4D online course demonstrated that a low-cost, open source software delivery platform combined 
with open educational resources (OER) could be used effectively to provide a hybrid MOOC environment 
that served over 1,400 learners. 
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Introduction 

BACKGROUND TO THE REPORT 

The concept of massive open online courses (MOOCs) has been hailed as both a disruptive change in higher 
education practice and an opportunity to provide the under-served with free access to university level 
courses. Making post-secondary educational experiences available at scale has been promoted as a 
opportunity for anyone with Internet access who is motivated to invest time in learning activities delivered 
in short video segments supplemented with online tutorials and assessed using automated online self- 
evaluation exercises and summative quizzes and tests. Khan Academy (2014) has popularized the format 
in K-12 education, while universities such as MIT and Stanford have demonstrated similar approaches for a 
higher education audience. 

Many MOOCs share a common goal of bringing large numbers of learners together (generally 1,000 
students or greater) in a common environment for a course delivered as a set of online lessons. However 
MOOCs are not all presented in the same manner and often differ in terms of their primary strategy for 
learning design. Some MOOCs use a traditional lecture format that substitutes video lectures for the face- 
to-face lecture component of a course and provide automated exercises and quizzes along with 
opportunities to interact among fellow students, and with the course instructors, using discussion boards 
or chat functions. This style of MOOC is often referred to as an xMOOC, and is characterized by the offerings 
from MOOC providers such as Coursera and Udacity and their university partners. These MOOCs tend to use 
custom-designed technical platforms, scheduled learning events, and proprietary learning resources. 

Another model of design is referenced as a cMOOC, and these systems use constructivist or connectivist 
pedagogies (Siemens, 2008) wherein learners are encouraged to build their own knowledge through social 
learning processes within flexible timeframes, guided by course instructors, and often using open 
educational resources (OER). It has been argued that the pedagogy of xMOOCs is better suited for learning 
domain knowledge that can be mastered through repetitive practice, and that cMOOCs may be better suited 
for allowing learners to acquire higher order creative skills (Bates, 2012b). The cMOOC concept was 
derived from course experiments led in 2008 by Canadian educational researchers, George Siemens (2008) 
and Stephen Downes (2006) that generated discussions about the pedagogical theory of connectivism, that 
proposed learning as a creative and social process of connecting nodes of knowledge. 

There are also hybrid design models such as those used by MIT and Harvard that use the edX open source 
technical platform. edX combines many of the key features of an xMOOC course delivery format and also 
uses open educational resources (OER). M4D was primarily a hybrid MOOC, similar to edA-based MOOCs. 

This report provides a pedagogical review of the Mobiles for Development (M4D) online course that was 
conducted over six weeks beginning October 2, 2013 by HT Kanpur and Commonwealth of Learning. It is 
intended that the review of the M4D course design and delivery experiences will help COL staff to better 
understand the prototype course, its participants, their performance, and their responses to the course 
design. 
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CONTEXT FOR THE M4D COURSE 

Currently, the use of MOOCs is being tested in the context of expanding capacity to meet the growing 
demand for educational and learning content and services worldwide. Many of the world's leading higher 
education institutions are attempting to provide access to high quality education for people around the 
world, including the disadvantaged and under-served. In this context, and drawing upon its experience in 
offering online courses on large scale, and its interest in fostering new perspectives on MOOCs in the 
context of learning for development, the Commonwealth of Learning collaborated with the Indian Institute 
of Technology Kanpur (IITK] to offer a six-week MOOC starting October 2, 2013. 

THE M4D COURSE DESIGN 

The topic of the M4D course was Mobiles for Development (M4D). Mobile devices have spread on a very 
large scale in developing countries and they have widespread presence, even in remote areas and among 
people with little or no history of using information and communications technologies (ICT], or 
telecommunications services provided through mobile phones or tablet devices. 

Course goals 

A primary goal for the M4D course was to determine how best to use MOOCs to serve the development 
needs of resource-poor communities of learners — those at the "bottom of the pyramid" (Prahalad & Hart, 
2002 ], 

To do so, the team conducted the M4D course with a view to better understanding the dynamics 
surrounding the design, delivery and support issues in delivering the MOOC-style courses in order to 
develop solutions and action plans that would support its primary goal for the course and lead to 
improvements in practice in subsequent iterations. 

Course team 

The course team included instructors, resource persons and mentors, and technical staff. Instructors, 
teachers and mentors were from IIT-Kanpur, IIT-Ropar, COL, Athabasca University, and the National 
Institute of Bank Management [India], 

The following were the M4D MOOC design team members. 

• Professor T.V. Prabhakar, Department of Computer Science and Engineering (IIT-Kanpur] was the 
principal instructor of the course. 

• Other members on Professor Prabhakar’s team included: 

o Dr. Balwinder Sodhi (IIT-Ropar] and Dr. Ashish Agarwal (IIT-Kanpur], 
o Dr. V. Balaji, Dr. K. Balasubramanian and Dr. A. Umar from COL served as resource persons 
and were supported by Mr. Ricky Cheng, a specialist in cloud services. 

Guest resource persons were featured in video-based case studies of practice from multiple regions of the 
world. 
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Some of the instructors gave time to support learner-mentor interactions in the online chat room and 
discussion forums. Four real-time chat sessions were organized to enable learners to chat (in text) with the 
instructors online and 18 time zones were covered during the chat sessions. 

COURSE AIMS AND OBJECTIVES 

The course instructors provided a clear set of aims and objectives for the course when it was first 
advertised to students through the Commonwealth of Learning, and using social media promotional 
techniques in networks that would attract participation in the developing world. 

The following seven text sections outline the course aims as stated by the course development team 
(Commonwealth of Learning, 2013a) and are directly cited from the website, Mobiles for Development: 
Massive Open Online Course (MOOC) and retrieved from http: //m4d.colfinder.org . 

About 

Mobile devices, phones as well as tablet computers have been spreading fast across the globe. The spread 
of mobile phones in developing countries has been increased dramatically. More people have access to cell 
phones today than to clean water. 

Mobile communication and computing technologies have much potential to contribute to human 
development. This is already felt in some measure in learning. There are reports of new advantages and 
benefits in other sectors such as agriculture and food production, and rural credit and finance. This course 
is about important concepts and practices in mobile technologies that are relevant to learning and 
education, agricultural extension and rural credit and finance. 

Outcome for Learners 

At the end of the course, learners will be able to recognize frequently used terminology, and identify 
techniques and methods in mobile computing and communications that are relevant to development. 
Learners will also be able to recount key use cases and successes, and recognize common policy 
requirements for the support of mobiles for development. This course is taught in English. 

Who Should Attend This Course 

This course would be of particular benefit to: 

• Teacher educators 

• Administrators in ODL organizations 

• Banking personnel in loan appraisal, and rural and agricultural credit 

• Managers of micro-finance organizations and community banks 

• Agricultural extension personnel in public and private sector 

• Civil servants with interest in e-Governance 

• Mobile tech developers 

• Students in ICT for development courses 
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Learner certification 

The Indian Institute of Technology Kanpur and Commonwealth of Learning (COL) will jointly issue 
certificates of participation to learners upon successful completion of the course. 

Requirements for participation 

This course is open and free to any learner. There are no pre-requisites. Any interested individual may 
register and participate. 

Commitment and support 

The expected commitment of learners is 4 hours/week on average. Instructors and mentors will be 
available online through the duration of the course to discuss topics and issues. 

Institutions 

The Indian Institute of Technology Kanpur is a premier research university that is pioneering a number of 
advances in mobile technologies and their applications. Notable developments in this area include the 
vKVK, a platform that connects KVKs with farmers through Internet and mobile technology, and the Digital 
Mandi initiative linking farmers and markets. IITK is also a pioneer in offering MOOCs. 

Commonwealth of Learning, an inter-governmental organization in the Commonwealth family, is globally 
known for its efforts and advocacy to improve access to learning at all levels. COL and partners in sub- 
Saharan Africa and South Asia have been making use of mobile technologies to help disadvantaged 
communities access learning and credit services. 

COURSE TOPICS 

The focus of the M4D MOOC was on helping learners increase their awareness of important developments 
in technology, essential terminologies, and the critical processes for engaging and working with 
stakeholders in this sector. 

Topics covered in M4D MOOC were: 

• The extent of the use of mobile technologies in a development context. 

• Terminology, techniques and methods in mobile computing and communications applied to 
development. 

• Information and case studies on mobile devices and technologies in education, rural banking and 
finance and agricultural extension using cases front multiple regions of the world. 

• Specific case studies where mobile computing and communication technology have been applied 
with some success in education, rural banking and agricultural extension. 

• Policies to support the use of mobile technologies for development. 
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Course structure and technical requirements for participation 

The course was structured around core topics and specialized modules related to human development. 

Learners were encouraged to complete all topics and modules, with the expectation that a motivated 
learner would spend about four hours on average per week. Instructors and mentors were available 
throughout the course to discuss topics and issues online in both discussion forums and chat spaces. 

As a MOOC-style course, there were no pre-requisites and access to the Internet was a requirement. 
Interested individuals could sign up for the course and learn any or all the topics, using desktop PC or 
laptop. Tablet and smartphone access was required, using devices running Android 2.2 and above, to 
participate in the M4D course. 

Course delivery platform 

The IITK team identified Sakai, an open source learning managements system (LMS) as the most suitable 
platform for use as an online classroom in the M4D course. Sakai was also mobile-enabled which provided 
additional flexibility and relevance to the topic, enabling interested learners to use tablets or smartphones 
to participate fully in the course. 

Course materials and resources 

All the learning materials were delivered as YouTube videos, and no video was longer than 25 minutes. 

PowerPoint slides were used and scripts for many of the videos were made available as PDF documents. All 
these materials were released as OER. The team considered the provision of these materials for download 
to be consistent with the M4D team’s advocacy of OER for development. At the time of this report, the M4D 
course was also one of a minority of MOOCs to make content available as OER. 

Assessment 

Two quizzes were provided, one at the end of main technology topics, and another towards the close of the 
course. 


Page 7 




M4D: MOBILES FOR DEVELOPMENT REPORT 


Approach to the design review 

Commonwealth of Learning staff requested an external review of the M4D MOOC. 

This review of the M4D course design and delivery experience was designed help COL staff to better 

understand the prototype course, its participants, their performance and their response to the course 

design. 

The following aspects were reviewed and reported and discussed in this report. 

• Course aims and objectives 

• Course design, resources, activities and evaluation strategies (pedagogical elements) 

• Course technical platform 

• Course results 

• Course cost analysis in relation to potential and/or actual outputs/ outcomes, and as compared to 
available benchmark figures for similar online courses (assuming cost figures that represent total 
cost of operations could be obtained conveniently from 11T Kanpur and COL) 

• Feedback front students, specifically a high-level thematic analysis of student comments, coded for 
anonymity 

• Feedback from instructors and staff through responses to written questions, with the assurance of 
anonymity for all respondents through the use of response coding 

REVIEW PROCESS 

The review of the course used the M4D websites, documents, questionnaire data, and existing literature. 

File and document review 

All documents and files for the M4D course were available online for review, primarily from public 

websites, but in the case of the course itself, an archive was used for retrospective review. 

Websites, files and documents reviewed were obtained from the following primary sources: 

• Commonwealth of Learning: MOOC for Development (M00C4D) 
http://www.col.org/progServ/programmes/KM/Pages/M00C4D.aspx 

• Commonwealth of Learning: MOOC on Mobiles for Development 
http://www.col.org/progServ/programmes/KM/Pages/M0QCM4D.aspx 

• Facebook: MOOC on Mobile for Development 
http://www.facebook.com/events/141746969368053/ 

• Mobiles for Development: Massive Open Online Course (MOOC) by IIT Kanpur and COL 
http://m4d.colfinder.org 

• Mobiles for Development: Massive Open Online Course (MOOC) by IIT Kanpur and COL 
http: //www.m4d-mooc.org through an unpublished archive to M4D course site 
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Questionnaires 

In addition, two surveys (questionnaires] and their results were reviewed. They were: 

• Questionnaire responses from 208 students who participated in the M4D MOOC 

• Questionnaire responses from 5 members of the M4D MOOC development team 

The student survey collected responses voluntarily and anonymously from students at the conclusion of 
the M4D course. The responses were collected online and exported to an Excel spreadsheet format for 
review and analysis. The data are presented and discussed in the Findings section of this report, which 
follows. 

Questionnaire responses from development team members were collected in March 2014, four months 
after the conclusion of the M4D course. The questionnaire was conducted voluntarily and anonymously 
online, using FluidSurveys.ca. The data are presented and discussed in the section of this report on 
Findings. 

Literature 

Current academic literature concerning the design and delivery of M OOCs was consulted and referenced as 
appropriate within the report. 

Literature on online course design, delivery and evaluation was also consulted and referenced as 
appropriate in the body of the report. 

LIMITATIONS 

Retrospective review of the M4D course 

The reviewer did not participate in the M4D MOOC during its delivery timeframe in October and November 
2013. As a consequence this review is retrospective, using available documents and course archives. No 
sense of the rhythm or ethos of the M4D course during its delivery timeframe was possible. 

Student questionnaire 

208 students provided responses to the voluntary questionnaire. The response rate can be calculated as a 
percentage of active participants who visited the M4D web site. Active participants numbered 1,441. 

• Active participant questionnaire response rate was 208 / 1,441 = 14 percent 

Instructor questionnaire 

Seven members of the design team were sampled using a questionnaire that asked questions about the 
learning design intent for the course from their personal perspectives. 

• Response rate from the design team sample was 5/7 = 71 percent 


Page 9 




M4D: MOBILES FOR DEVELOPMENT REPORT 


Review findings 

FRAMEWORKS FOR ANALYSIS 

Bates and Poole [2003] provide useful frameworks for evaluating effective teaching with technology in 
higher education. In particular Bates used the acronym SECTIONS to describe one specific model that is 
useful in the context of the M4D course review. It is described in the table below. 


SECTIONS FRAMEWORK (BATES & POOLE, 2003) 

SECTIONS THEME 

ATTRIBUTES 

Students 

Student demographics, needs, situation 

Ease of use or access 

Ease of use and reliability, 

Cost 

Cost considerations for students and institution 

Teaching functions 

Instructor’s approach to Teaching and learning 

Interaction 

Desired level of Interaction for students 

Organization 

Organizational support needed 

Novelty 

Novelty factor 

Speed 

Speed with which the technology can be adopted or 
implemented 


Bates has updated his framework over the past few years to account for new technologies and models of 
practice, such as MOOCs. Currently he is discussing a more refined schema in his presentations and blog 
posts, and according to Bates (2014a) the new schema (PSLR) will be useful for "deciding where in the 
continuum of online learning a course or program should be, but it doesn’t have a catchy acronym.” 

This updated schema asks four questions that educators and instructional developers should consider 
prior to designing a new course or program (Bates, 2014a): 

• Pedagogy: How would you really like to teach (e.g. information transmission; experiential; inquiry- 
based; knowledge management)? 

• Students: What kind, what age, ability to learn independently; experience in the subject area? 

• Desired learning outcomes (skills and content) to decide what is best done online - or the 
requirements of the subject discipline? 

• Resources: What instructional design and technology support is available? 

The PSLR framework and elements of the SECTIONS framework were used in reviewing the pedagogical 
and technical approaches used by the M4D team. In particular, the PSLR framework was used to develop 
questions requiring a text-based response from the M4D development team in the March 2014 online 
questionnaire. 
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COURSE DESIGN 

The M4D course was delivered using the Sakai open source technical platform. The screen layout for the 
course was designed for straightforward navigation to all components of the course. The features of the 
M4D course are numbered in the screen capture below, and are described in the legend below the screen 
capture. 
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Features of the M4D Sakai course environment: 

1 Course header and student workspace 

2 Navigation menu for the course 

3 Topic list 

4 Weekly module dividers 

5 Colour-coded topics 


6 Login and course role graphic 

7 Color-coded legend for topic 

8 Instructor for topic 

9 Video link for topic 

10 Slides and/ or script link/ s 
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COURSE STRUCTURE 

Resources 

Resources were comprised of the following media types: 

• YouTube videos of lesson topics and case studies 

• Scripts to accompany many of the video lesson topics 

• PowerPoint slides to supplement lessons 
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The primary instructional strategy for the M4D course was the use of instructional videos that varied in 
length from 2-25 minutes. 

In total there were 92 videos produced by the development team for the topics for the course. The videos 
were organized over a six-week time period. Students were required to view 15 videos per week on 
average. 

Most video lectures also included a supplementary PowerPoint slide deck. In some cases, a script was also 
produced to accompany video, and student feedback indicated that scripts were helpful in dealing with 
unfamiliar accents or speech intonation. 
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M4D Course Activities 

There were two activity strategies employed during the course: 

• Chat room 

o 1,641 messages were exchanged in the Chat Room during the course 



• Discussion forums 

o General discussion forum with 398 messages across 76 topics 
o Technical forum with 370 messages across 55 topics 
o Technical support forum with 89 messages across 35 topics 
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Evaluation strategies 

Online quizzes were employed as an evaluation strategy. There were three quizzes during the course: 

• Test Quiz 324 students submitted quizzes for evaluation 

• Quiz 1 296 students submitted quizzes for evaluation 

• Final Quiz 261 students submitted quizzes for evaluation 

The quiz format used multiple-choice questions as illustrated in the screen captures below from the Final 
Quiz of the M4D course. 
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COURSE RESULTS 

The course began with 2,282 course registrants, from which 1,441 registrants were considered active 
participants in the course during its six-week timeframe. Site statistics from the M4D course site provide a 
snapshot of activity and participation. 

Students 

Although no detailed individual demographics are yet available for the course participants, the following 
summary data provides a high-level snapshot of course participants and their participation level: 

• The course attracted 2,282 registrants from 116 countries. 

• About 63 percent of the registrants (1,441) were active in the course space. 

• A total of 333 participants were found eligible for receiving certificates of Competence (244 
participants) or Participation (89). Certificates were awarded to approximately 23% of active 
participants in the M4D course. 

• The top five countries in terms of registrants were India, Nepal, Mauritius, Grenada and South 
Africa. 

• About 500 registrants were from countries in the Africa-Caribbean-Pacific regions. About 200 
registrants were from the OECD countries and from East Europe. 
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For the participants who participated in the M4D course quizzes, two kinds of certificates were awarded. 

If a participant performed well in the quizzes, the instructional team awarded her/him a "C" (Competency) 
certificate. For those students that did not do as well in quizzes, but had participated in the course to a 
reasonably consistent level, the team awarded a "P" (Participation) certificate. An M4D course participant 
was awarded only one type of certificate. 
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Precise criteria for both of the "C" and "P" certificates is described below: 

• If a student’s aggregate score front all quizzes was more than 40%, then the student was awarded a 
"C" certificate. For example, assuming both quizzes to be of 100 marks and the student scored 20 
marks in quiz-1 and 70 in quiz-2 then the combined score (20 + 70 = 90) is more than 40% (0.4 x 2 
x 100 = 80), hence she/he would be awarded a "C" certificate. 264 “C” certificates were awarded. 

• If the student did not qualify for a "C" certificate, but her/his aggregate score from "attendance" 
and quizzes was more than 50%, then the student was awarded a "P" certificate. The instructors 
calculated the "attendance" score by statistics for Core Lecture videos/slides (PDF). For example, if 
a student’s score on "attendance" was 45% and her/his score on all quizzes was 5% then the 
student was awarded a "P" certificate. 89 “P” certificates were awarded. 

Based on these criteria, certificates were prepared. Each certificate, "C" or "P" was jointly signed by the 
Center for Continuing Education, I1T Kanpur and the Commonwealth of Learning. Certificates were 
awarded to 333 students, approximately 23% of active participants in the M4D course. 
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COURSE TECHNICAL PLATFORM 

Background to the technical platform selection 

While the M4D course team was planning the M4D course, it tested the software platform Canvas, as a 
hosted service from Instructure (the vendor), to determine its feasibility as a technical system for course 
delivery. Cost estimates provided by Instructure for the hosted service were as follows: 

• The quoted cost for 400 learners was USD $10,000 for six weeks 

• Additional students up to 1000 (that is, an additional 600) were to cost $3.28 per "seat” for this 
period 

• The total cost estimate for 1600 students was: $10,000 + (600 X $3.28) = $11, 968. 

• There was no clarity on numbers beyond 1000 additional students 

The vendor offered the following standard client services as a part of the proposed contract package: 

• Authentication integration assistance for supported identity providers, including LDAP, CAS or 
SAML 2.0 

• Standard Online Training Package that included administration, user support, and instructor 
training webinars 

• Site branding assistance with colour header for website with a logo for the institution 

• Basic student information system (SIS) endpoint, with a bulk enrollment API enabled that 
accepted updates provided in the Canvas SIS import format 

• Test/training instance: refreshed every three weeks (to coincide with the Canvas release schedule) 
with production data for the duration of the account 

• Premium support package (details unspecified) 

• The vendor indicated that the implementation process would take 90 days from contract signing to 
execution of a live M4D course site 

Because the Instructure proposal appeared too costly, the team proceeded to use the complied open-source 
"community edition” software code for Canvas, available from GitHub, and installed the code on the Amazon 
cloud service. However, it became quickly evident to the development team that open source edition of the 
Canvas code presented serious stability issues that would be unacceptable as an implementation platform 
for the M4D course. 

The choice of Sakai as the technical platform for M4D 

As a consequence of the testing of Canvas, the team decided immediately, and one week prior to M4D 
registration opening, to switch to the open source Sakai platform. The team at 11TK wrote a custom set of 
software code in a short timeframe in order to make Sakai compatible with the registration interface 
previously built on the Drupal platform. 

Sakai was chosen as the course technical platform because it was available as open source software and 
had a reputation for scalability with large numbers of students. This proved to be the case in the M4D 
course. 
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The IITK team identified Sakai, an open source platform, as most suitable for use as an online classroom in 
this course. It was also mobile-enabled so that some of the interested learners could access it from tablets 
or smartphones. 

An external review of the Sakai software describes some of the technical platform’s key features, many of 
which were used in the M4D course (University of Virginia, 2014): 

"The Sakai software includes many of the features common to course management systems, 
including document distribution, a grade-book, discussions, live chat, assignment uploads, and 
online testing, but the capacity for online collaboration is really the tail that wags the dog here. 

Sakai is intended as a collaborative tool for research and group projects. To support this 
function, Sakai includes the ability to change the settings of all the tools based on roles, 
changing what the system permits different users to do with each tool. It also includes a wiki, 
mailing list distribution and archiving, and an RSS reader. 

If online student collaboration, within classes or not, is a major part of what you will be doing 1 
strongly recommend trying Sakai. Trying Sakai is low risk because it is free and you generally 
can demo it at www.Sa/m/Project.org without having to install it. You can download the 
software to install at that same URL. " 

The Sakai technical platform is highly capable and scalable and proved to be a stable technical platform on 
which to mount and conduct the M4D course with approximately 1,400 students. 

However, it is a plain and simple learning system in its raw form and could use graphic design support to 
effectively brand it, and increase its eye appeal for learners. The additional of graphic design elements, 
institutional colour schemes and logos could require additional cost by the development team in future. 
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COURSE COST ANALYSIS 

One possible framework for cost analysis 

Bates (2001) provides a sample schema for analyzing costs for an online course or program development 
process that includes the following elements that require cost analysis. 

• Planning 

o Including time and effort costs required for meetings and development discussions 

• Program administration and overheads 

o Including time and effort costs required to use institutional systems and processes for the 
development, delivery, assessment, and certification of the course 

• Development 

o Including time and effort costs for faculty to develop content, media to be produced and 
mounted on the online course delivery system 
o Including time and effort costs for custom software development 

• Maintenance 

o Assuming the course will be maintained and delivered again to help amortize overall 
development costs 

• Delivery 

o Including license, time, and effort costs associated with the operation of the technical 
platform on which the course was delivered 

Intangibles and variables in the model include the calculation of faculty costs in the development and 
delivery process. However, ‘targets’ are often set for faculty workload and are included in their tenure 
costs. In the Bates model the aim was to make sure the teaching load for a fully online course was no more 
than that of a face-to-face course for a tenured professor. The target was also influenced by experience from 
earlier online courses concerning the faculty time needed for these kinds of courses. 

COL cost analysis 

From documents provided to COL by IIT Kanpur the following conclusions were drawn: 

• COL provided a total of C$15,000, which was converted into Indian Rupees. 

• The bulk of the course costs (just over 75%) were incurred in content development, including 
costs of recording and editing of instructional videos. 

• The IITK team provided support for server management costs, which was actually a substantial 
contribution although they, as a public institution, had not placed a monetary value on IT support 

In terms of COL’s contribution, the Director, Technology & Knowledge Management contributed about five 
percent of his time of his time to the project and COL hosted the home page of the course, 
http://m4d.colfinder.org on its server at an annual hosting cost of USD $120. 
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Other NT Kanpur costs 

The IITK team had full administrative privileges on the http://m4d.colfinder.org server and the 11TK team 
added to this a secure and reliable registration system from which they transferred data in a secure way to 
the online class site, www.m4d-mooc.org . 

The transfer process was developed exclusively for this course by IITK, since it involved transfer from 
Drupal to Sakai for which there are no known solutions. 

This has software development process has not been assigned a cost by IITK. 

Reviewer comments 

A clear need for future iterations of the M4D course and similar courses that may be considered for 
development by COL and IIT Kanpur, is the need to use a consistent cost-analysis framework that would 
account for all costs. 

Based on the available data for the M4D course, the investment of C $15,000 and the USD $120 hosting 
costs for the http://m4d.colfinder.org server were the only tangible costs that could be provided for this 
review. 

Bates (2011) noted that faculty time and effort required for development of materials and instruction tend 
to be the major costs required for online course delivery. Therefore the total costs for development and 
delivery of the M4D course could be estimated by assigning an hourly rate to the efforts of faculty, media 
staff and technical staff. Using this method and an estimate of hours invested by each would provide an 
approximation of the total cost for development and delivery of the M4D course. These costs could be 
estimated retrospectively to produce a benchmark metric for future development. 

The literature on MOOC development provides estimates of course development and delivery costs 
between USD $50,000 and $250, 000 for a single instance of a large-scale MOOC. These cost estimates 
suggest that even with limited cost data available that the M4D course would likely be at the low end of the 
cost scale for development and delivery. 
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FEEDBACK FROM STUDENTS 

An online student survey was conducted with students after the course was completed. A total of 208 
responses to survey questions were received from students. For each question students were provided 
with a five-point scale with which to provide a numeric answer, with 1 being the lowest rating and 5 being 
highest rating. The questions, the student ratings, and averages for numeric ratings are provided in the 
tables below. 

Overall student satisfaction based on response to questions with numeric responses was 87 percent, 
indicating strong satisfaction for the instructors, course content, resources, and delivery format. 


STUDENT SURVEY RESPONSES TO QUESTIONS REQUIRING A NUMERIC RATING 


QUESTIONS 

NUMBER OF RESPONSES 

AVERAGE RATING 

The website was easy to use. 

208 

4.38 

How would you rate the usefulness of 
discussions on forums/chat sessions? 

207 

4.06 

Questions in the quizzes were relevant 
and well chosen. 

205 

4.12 

How was the clarity of instructor's 
presentation of the material (slides, 
audibility, vocal clarity)? 

205 

4.30 

The short course was delivered consistent 
with its stated objectives. 

207 

4.40 

Overall 1 am satisfied with the quality of 
this short course. 

208 

4.45 

The presenter had good knowledge of the 
subject content. 

206 

4.66 


The student survey asked two questions requiring YES, NO, CAN’T SAY responses. The questions and 
responses are detailed in the table below. 71 percent of students indicated they would have taken the M4D 
course even without certification. 97 percent stated they would recommend the M4D course to others. 


STUDENT SURVEY RESPONSES TO QUESTIONS REQUIRING A YES, NO, CAN'T SAY RESPONSE 

QUESTIONS 

YES 

NO 

CAN'T SAY 

Would you have taken this course if there 
were no certificate offered? 

147 

31 

29 

Will you recommend this course to others? 

201 


7 
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In addition to questions requesting numeric ratings and Yes/No responses, students were provided four 
open-ended questions that requested a text-based response. The text-based responses from 208 students 
were coded and clustered as themes by the reviewer, using qualitative analysis software. The questions and 
the emergent themes from student text-based responses are detailed in the tables that follow. 

The top five emergent themes from the coded and clustered responses are listed for three of the questions. 
For the fourth question, asking for interest in other course topics, an expanded list of themes is provided. 


STUDENT SURVEY RESPONSES TO QUESTIONS REQUIRING A TEXT-BASED RESPONSE 


QUESTION 


EMERGENT THEMES FROM RESPONSES 


What did you particularly like about this 
short course? 


What suggestions do you have on how we 
can improve this short course; in its 
content, delivery or administration? 


• Course content, and especially its practicality 

• Course format and especially its use of videos 
with scripts, slides, case studies and online 
quizzes 

• Relevance of the topics and case studies to the 
developing world, agriculture and banking 

• Professional knowledge of the instructors, 
their preparation, and the quality of 
explanations of the course topics 

• Convenience of the online course format and 
the flexibility of the design to accommodate 
students’ working lives 

• Video: Shorter and higher quality video and 
audio sequences of between 8-12 minutes. The 
ability to more easily download the video 
segments and case studies for offline viewing. 
More uniform audio quality throughout - some 
were of lower quality. 

• Practicality: More use of applied topics, 
mention and use of mobile applications, and 
practical examples and demonstrations were 
requested by some students who found the 
topics too technical for non-technical 
participants 

• Assessment: More quizzes, perhaps on a 
weekly basis at the end of topic segments. The 
ability to re-do quizzes to achieve mastery. 
More variety in the style of online quizzes 

• Discussion summaries and topic summaries 
provided on a weekly basis 

• Hands-on, live or practice exercises that 
required students to practice their learning 
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What did you NOT like about this course? • Length of videos presented access problems for 

live viewing in areas with Internet connectivity 
issues. Consider download or pre-packaged 
videos on CDs or DVDs as alternative formats 

• Too technology focused, with a need for more 
practical examples and case studies 

• Intensity of the course and heavy video content 
load made it difficult for late starters to catch 
up, or working students to catch up if they fell 
behind 

• Quizzes had too many technical questions for 
non-technical participants 

What other short courses would you be • Educational topics: instructional design, 

interested in for the future? applications of mobile technologies for 

teaching and for use in libraries 

• Agricultural topics: GPS, GIS, meteorology, 
fuzzy logic, mobile use in agriculture with 
expanded cases studies from other areas of the 
world 

• Wireless and network topics: mobile 
application development especially for Android 
devices, cloud computing, web analytics 

• Security: cyber-security, ethical hacking 

• Management: knowledge management, 
management skills, technology transfer 

• Research: research methods, research methods 
employing mobile technologies 

• Digital media: web development, media 
production, HTML5, media storage 
preservation 

• Entrepreneurship: small business 
development, small business development in 
rural settings 

• Health: applications of mobile technology for 
health and safety applications 

• Finance: banking, alternative banking systems, 
micro-finance, organization and management 
of cooperatives 

• Gender: gender equity, gender issues 


The open-ended responses indicated strong support for the M4D course. The responses also provided an 
opportunity for students to highlight areas for improvement in subsequent iterations of the course. The 
feedback was both positive and constructive, and should be incorporated into future course designs. 
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FEEDBACK FROM THE DEVELOPMENT TEAM 

An online questionnaire was provided to seven members of the M4D MOOC development team members in 
March 2014, four months after the conclusion of the course. The questionnaire asked team members to 
consider the course from a design perspective, and in particular from their roles as team members. 

Five team members provided responses to nine questions that included eight open-ended text-based 
response questions, and one question that required a checklist response. The responses are provided in the 
sections that follow, along with some discussion of the responses from the reviewer’s perspective. 


QUESTION 1 



QUESTION 

DEVELOPMENT TEAM RESPONSES 

What was your role in the M4D MOOC 

• 

1. Instructor (course design and content 

course design, development or delivery 


creation) 2. Technology expert for MOOC 

processes? 


platform 


• 

Played a role in all parts - conceptualise, 
design, develop and delivery 


• 

Instructor 


• 

I helped in the development of the course by 
creating 8 videos that were used in the 
course. 1 was also a moderator for a 
synchronous session and I participated in the 
asynchronous forums. 


• 

I was mainly responsible for setting up and 
managing the technical platform, content 
release process and delivering technical 
lectures on Android, Mobile OS and App 
Development. 


Reviewer comments: 


Each of the participants who responded indicated a clear understanding of her/his role within the M4D 
course team. 

Experienced educators with a background in open and distance learning (ODL) might be expected to 
understand the differentiated staffing models typical of ODL institutions and programs. In the case of the 
M4D course, the experience of the individual team members was revealed through the clarity with which 
each of those professionals specified her/his role within the team in the questionnaire responses. They 
knew their roles and executed them within the M4D course framework. 

As a general recommendation, the selection of participants for MOOC design, development or delivery 
processes would benefit from having individuals with significant ODL experience as a part of the team. 
MOOCs are complex educational projects requiring defined instructional development and delivery skill 
sets. 
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QUESTION 2 


QUESTION DEVELOPMENT TEAM RESPONSES 

What were your personal goals associated • Wanted to share my knowledge in the domain 

with participation in the design, and learn about how to 'run a MOOC' - end to 

development and/or delivery of the M4D end. Also wanted to explore platforms for 

MOOC? That is, what did you hope or delivering a MOOC. 

expect to achieve through your • share my knowledge of mobile learning and 

participation in M4D? participate in an m-learning dialogue; guide 

students; support the m-learning community 

• (These points are not in any particular order.) 

1. Knowledge dissemination. 2. Gain more 
experience with new delivery model (i.e. 

MOOC) for knowledge dissemination. 3. 

Showcase and attain greater visibility for our 
capabilities for end-to-end design and 
operation of MOOCs. 4. Generating new 
knowledge/ideas via online discussions that 
happened over this MOOC's forums. 5. 

Validate our costing models for end-to-end 
design and operation of the MOOCs of this 
nature. 

• I believe that learning materials should be 
available and made affordable to people in 
developing countries. This was my personal goal 
in getting involved in the MOOC course. 1 am also 
interested in MOOC research and this project gave 
me ideas on research areas that are needed. 

• I was interested in broadening my knowledge 
in the space of M4D by learning from other 
instructors and students. 1 was also 
interested in gaining practical experience of 
managing a MOOC platform, understanding 
issues in running a MOOC like generating 
good quality content, etc. 

Reviewer comments: 

The statements of participants about their personal goals for M4D were highly congruent with stated aims 
and objectives of the course, and the partnership between ITT Kanpur and COL 

Not only did team members express a strong desire to provide accessible learning at low cost, they also 
expressed a willingness to both learn and share knowledge, and to become co-learners with students and 
other instructors. In particular, team members expressed a desire to learn more about MOOCs as a delivery 
format and as a pedagogical practice, and through the course contribute new knowledge. 
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QUESTION 3 


QUESTION DEVELOPMENT TEAM RESPONSES 

Which teaching or pedagogical strategies • Major pedagogical strategies supported by 
do you believe were supported within the M4D in that order: 1. Information transmission 

M4D MOOC (for example: information 2. Inquiry-based learning 3. Collaborative 

transmission, experiential learning, learning 

inquiry-based learning, collaborative • Information transmission - is the major one. 

learning, knowledge management]? Experiential learning and inquiry-based 

learning were handled through homework (not 

Comment on any that you feel should be graded - hence not known if the student has 

added or elaborated. done them], 

• Information transmission, collaborative 
learning, knowledge co-creation through 
dialogue 

• There was information transmission in the 
asynchronous session where participants were 
able to view videos, access the PowerPoint 
presentations, and read the transcripts of the 
videos. This MOOC was flexible since 
participants who were not able to watch the 
videos because of low connectivity, were able 
to download the PowerPoint slides and the 
video transcripts to read at a later time offline. 

The MOOC allow for participants to connect 
with each other using forums. The forums were 
very active. In addition to posing questions, 
participants provided information on their 
own experience with mobile for development. 

One suggestion is that there should be sub- 
forums where groups work together on 
problems and projects and share with other 
participants. There could be more problem- 
based activities. 

• I believe that M4D MOOC supported many 
teaching strategies like learning material from 
field experts (video lectures/slides] were 
following information transmission strategy. 
Participation in forums and chat rooms 
provided collaborative learning among 
students. A very good quality of content has 
been generated in forums and chat rooms. 

Reviewer comments: 

Team members’ responses indicated that the primary strategy was information transmission in the design 
of the M4D course. Collaborative discussions contributed to knowledge building and better understanding. 
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QUESTION 4 


QUESTION DEVELOPMENT TEAM RESPONSES 


Please comment on the instructional 
design or technology support that was 
available to you as a developer or 
instructor for the M4D MOOC, and 
whether the support met your 
pedagogical objectives for the course 
and its students? 


Reviewer comments: 

Most of the team comments dealt with 
the technical aspects of instructional 
resource production. In some cases 
team members had knowledge or skills 
that enabled them to produce their own 
resources. In other cases specialized 
support was provided, particularly for 
video production. 

One role that was not discussed 
explicitly in comments was 
instructional design support for the 
primary structure of the learning 
outcomes, resources, activities and 
assessment strategies within the M4D 
course. This is an area of practice that 
also requires specialized support from 
professionals. 


Instructional design: It was mainly from my own 
experience; I did not look for any specific 
support in this area. However, I reviewed my 
lecture briefly with co-instructors to get their 
feedback for any glaring issues or improvements. 
Technology support: I myself was one of the 
technology support persons (There were others 
as well] for M4D MOOC. Tech support included: 

1. Managing content on the LMS platform that 
we used. 2. Monitoring the web server for 
security and other issues (usual stuff associated 
with any public facing web application], 3. 
Editing raw lecture videos. I believe that both 
instructors and students were satisfied with the 
generation and delivery of content (whether 
generated by instructors as video lectures, or by 
students via online discussions]. 

For creating content, we used a mash-up of tools 
readily available in the Institute - like recording 
studio, Adobe video editing, MS PowerPoint, 
IHMC Cmap tools. Do wish we had professional 
level video editing and animation support was 
available. 

Instructional design support provided during the 
video creation process was crucial to the success 
of my contribution to the MOOC; well-designed 
professionally looking multimedia materials are 
a vital element of any online course. 

The video production facilities at COL allowed 
me to develop the videos. The LMS worked well 
for the course. For MOOC in developing countries 
it is important to have simple LMS that allow 
people with low connectivity to access the 
learning materials. The LMS was appropriate for 
this. 

We used Sakai, which is an open source MOOC 
platform. From functionality point of view, it had 
most of the features available to us. However, a 
lot of effort has been spent in customizing these 
features according to our needs. For course 
registration process, we developed a new Drupal 
based application. 
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QUESTION 5 


QUESTION 


DEVELOPMENT TEAM RESPONSES 


Please comment on the attributes of the 
technical platform used to deliver the 
M4D MOOC and in particular its ability to 
support your pedagogical objectives for 
students? 


Reviewer comments: 


• Primarily we wanted to have two pedagogical 
threads: 1. For dissemination of content 2. For 
allowing collaboration amongst participants. 
Second one — collaboration — inherently 
allowed for inquiry based learning. Technology 
platform that we chose supported the above 
two threads (it had online discussion forums 
and chat rooms, and also well sequenced video 
lectures]. Other area where choice of 
technology platform was important was 
content creation. Chosen tools allowed us to 
record/edit video lectures in a flexible manner. 

• Used Sakai, hosted on Amazon. It is not a 
sophisticated platform like edX, but it suited 
quite well since our audience were not 
challenged by the platform. The chat room 
could have been better, though. 

• The selected platform offered an environment 
conducive to online learning through resource 
and expertise sharing. Materials were well 
organized and linked to the relevant discussion 
and private space in the platform. There were 
tools for synchronous and asynchronous 
interaction supporting knowledge co-creation. 
The platform was rather easy to navigate and a 
variety of tools were available to enable a more 
personalized learning experience. Students 
could easily contact other learners as well as 
instructors. 

• The platform allowed for the presentation of 
the information and for forum participation. It 
was simple for the novice to use. In this course, 
there was technical help for those with access 
problems. 

• The Sakai platform was robust and 
customizable. We had very few minor technical 
issues during the course. It had all features to 
handle our pedagogical objectives. 


As indicated by team responses, the Sakai platform worked well for the design and delivery of the M4D 
course and supported the two primary design needs for dissemination of content and support of 
discussions. Some refinements to chat spaces and discussion forums may be needed in future courses. 
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QUESTION 6 


QUESTION DEVELOPMENT TEAM RESPONSES 


Please comment about how well you believe 
the M4D MOOC design supported the 
desired learning outcomes that the course 
was intended to deliver. 


Reviewer comments: 

Team members took the opportunity in this 
question to comment about the success of 
the M4D course in achieving its desired 
learning outcomes, and to offer thoughts on 
how the M4D design might be improved. 

Improvement strategies suggested included: 

o Peer groups or TAs to support learners 
o More quizzes after each lecture 
o Lecture-based discussion threads 
o Personalization tactics 
o Activities to help develop skills 
o Opportunities for local sub-groups, 
support groups and access to locale 
experts 

The recommendations for improvement 
cited by team members in Question 6 should 
be used as a set of design features for 
consideration and testing in future iterations 
of the M4D course. 


Course participants, expectedly, had large 
variation in background and course access 
environment. Despite this diversity, the 
online discussions/chats that happened on 
M4D showed that the intended learning 
outcomes were achieved to a great extent. 
Correcting the homework (either through 
peer groups or through TAs) would have 
helped greatly. More quizzes, especially in 
the middle or immediately after each 
lecture would be good. Lecture based 
discussion threads (even though this was 
attempted, platform support was not 
available) would have helped. 

As mentioned above the MOOC design was 
effective; however, additional strategies to 
guide participants to the discussions and 
materials that were of particular interest to 
them would be desirable. For instance, 
forming sub-groups or discussion forums 
focusing on a particular issue/topic as 
opposed to having all discussions in one 
overwhelmingly large forum. Any strategies 
to provide more guidance towards 
resources/discussions on particular 
interests of learners, as well as 
personalizing the experience - making 
participants feel that they are individual 
student as opposed one of a few thousands 
of students. 

The learning outcomes of the course were 
more of an awareness of M4D and 
information on how to use M4D and case 
studies around the world. Hence, the design 
supported the learning outcomes. If the 
learning outcomes were to develop skills in 
M4D then more depth should be covered 
and the participants should be given 
activities to develop the skills. To develop 
the skills, a blended approach will be good 
where participants learn the information in 
the MOOC and they work with local experts 
to develop the hands on skills. 
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• The course was designed to teach technical 
concepts through real life examples and 
case studies. By looking at the participation 
in forums and chat rooms, I believe that the 
course did a great job in empowering 
participants (with little or no technical 
background] to use mobile technologies in 
development issues. 


QUESTION 7 


QUESTION 

Please comment on what you believe might 
be desirable attributes that students should 
possess to be successful in a course such as 
the M4D MOOC. 


Reviewer comments: 


DEVELOPMENT TEAM RESPONSES 

• 1. Should be comfortable and enthusiastic 
about online engagement. 2. Should have 
access to the Internet. 

• Engagement. Wanting to learn rather than a 
certificate (we had a few like that] 

• Commitment to participation and 
completion of the entire course or all the 
elements of the course that are relevant to 
them, which might be achieved by 
customizing the learning path for 
individuals in the course. 

• Students should have the following 
attributes: (1] good communications skills 
(2] good ethics not to offend anyone (3] 
basic computer skills (4] flexibility in time 
to access the course materials and to 
participate in synchronous sessions 
because of different time zones (5] good 
time management skills (6] be able to read 
and understand the language the language 
the MOOC is delivered in. 

• To be successful in a course like M4D 
MOOC, students should regularly cover the 
course material and actively participate in 
the course activities. Apart from learning 
content, the course provided a platform to 
students for discussion on development 
issues with the field experts as well as other 
students. 


Desirable attributes for students in a course such M4D included skills that might be used to describe 
independent learners or distance learners. Each team members described somewhat different attributes. 
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QUESTION 8 

QUESTION 

DEVELOPMENT TEAM RESPONSES 

Please identify the key features of the M4D 
MOOC, from your perspective. Using labels 
from the M«0«0«C illustration below as a 
guide, please make your choices from the 
checklist in Question 9, below. 

• No responses were required in Question 9. 
Questionnaire participants were asked to 
review the illustration in preparation for a 
self-assessment of the M4D MOOC’s 
attributes in relation to the illustration 
provided as an organizer for design 
attributes associated with MOOCs. 



FOCUS ON 
COMMUNITY 
AND CONNECTIONS 


What is 

massive? 

• 100? 

• 1 , 000 ? 

• 10 , 000 ? 

• 100 , 000 ? 


Local 

cohorts? 


Open 

registration? 


Self-paced? 



Start/end dates? 


College credits? 


MASSIVE OPEN ONLINE COORSE 

\ 

~ , \ Real-time 

Open content? / \ 

I \ interaction? 

Free of charge? 



Badges? 


Role of the 
instructor? 

Learning 

community? 


Scripted assessments 
and feedback? 


Affordable? 


Image by Mathieu Plourde (2013). 
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QUESTION 9 

QUESTION 

DEVELOPMENT TEAM RESPONSES 

Use the checklist to select what you believe 
to be features of the M4D MOOC, using the 
diagram in Question 8 above as a 
reference. 

• The graph below describes the scale of 
response on each checklist item. 


Response Count 

X - Focus of the design was on scalability of the environment 20 
C - Focus of the design was on community and connections 
Open registration with no stated pre-requisites 
Local cohorts (regional, state, city) as an organizational 


feature 

Self-paced learning 

Start/end dates for the course were stated 
Badges, or other certification offered 
Instructor mediation within the course 
Learning community provided for student support 


Automated or scripted assessments and feedback for 

40 . 0 % 

students 

Show more results... 




Total 


5 


Reviewer comments: 

The responses by the M4D development team are indicative of the hybrid MOOC environment that was 
created for the course. Although M4D registered over 2,000 students, of which over 1,400 were active 
learners in the course, the design intent of a majority of the team was to create community and connections 
as a part of the learning experience for students. In this sense, the M4D more closely resembles edX style 
MOOC courses than it does either the large-scale xMOOCs, or connectivist cMOOCs, where the curriculum is 
formed through student-instructor interactions with significant exploratory, social and experiential 
components. M4D sits midway along a continuum that has xMOOCs on one end and cMOOCs at the 
opposite end. 

Other comments from the development team questionnaires concerning recommendations for local 
support groups, segmented and structured discussion forums, and additional experiential and hands-on 
activities for students, hint at a desire to design a MOOC environment that moves closer to a cMOOC format. 
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QUESTION 10 


QUESTION DEVELOPMENT TEAM RESPONSES 

• None. 

• We send the course material to some of the 
students, on a CD. The course was about 
Technology for the non-technologist. It was 
about 'solution architecture' — a design of the 
solution by the Domain expert with 
'architectural level knowledge' of the 
technology (for example, missed calls and 
how to use them in a solution). 

• ffaving participated in a number of MOOCs, I 
believe that it would be beneficial to learners 
to provide them with a more personalized 
experience offering expert support in terms 
of guidance through the course materials and 
activities as well as the subject matter 
expertise. The role of the instructor and other 
learning supports remains vital which results 
in certain limitations in terms of the cost and 
start/end dates of the course. 

• For the first implementation of this MOOC, 
the design and delivery were excellent. For 
future delivery the following are suggested. 

(1) Have weekly forums (2) Get people in 
local areas to meet each other for 
collaboration (3) summarize the forums so 
participants get a sense of completion (4) 
provide high level discussion questions for 
critical thinking. 

Reviewer comments: 

Responses to the final question about design intent and design features reiterated some of the 
recommendations that have been noted in previous responses including: 

o Use of alternative resource delivery formats for student in situations where Internet connections and 
bandwidth issues were an impediment to participation in the course (i.e. CDs and DVDs) 
o Personalization of the learning experience to better meet the needs of students 

o Structured, weekly discussions forums 

o Engage students in higher-order critical thinking through discussion strategies 
o Discussion and/or topic summaries on a weekly basis 


Please provide additional comments about 
the design intent or design features of the 
M4D MOOC that were not addressed in the 
previous questions. 
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Summary and Recommendations 

M4D WAS AN INNOVATIVE ONLINE COURSE 

Bates (2014) cited a video presentation by Dr. Rory McGreal, Contact North | Contact Nord Research 
Associate and the UNESCO/Commonwealth of Learning Chair in Open Educational Resources, in which 
McGreal suggested some steps that can help faculty and instructors to approach the issue of innovative 
online teaching in a systematic way, including: 

• Being clear on the problem you are trying to solve 

• Working in a team 

• Applying technology appropriately to address the problem to be solved 

• Evaluating and disseminating your innovation 

• Providing greater flexibility for students as a new mantra 

It is clear from the review of the M4D course that its instructional team had intuitively followed the steps 
outlined by McGreal. A primary goal for the M4D course was to determine how best to use MOOCs to serve 
the development needs of resource-poor communities of learners — those at the "bottom of the pyramid" 
(Prahalad & Hart, 2002). Course outcomes and feedback from students indicated that an M4D goal was 
achieved, and that students were constructive with their feedback and suggestions for improvement. 

The team also conducted the M4D course with a view to better understanding the dynamics surrounding 
the design, delivery and support issues in delivering MOOC-style courses. Their aim was to develop 
solutions and action plans that would support the primary goals for the course and lead to improvements 
in practice in subsequent iterations of the M4D. 

It is in the spirit of the M4D team’s general goals that the following recommendations are offered. 

RECOMMENDATIONS 

Recommendation 1 

Adopt an iterative, design-based research (DBR) approach for all MOOC-style courses developed jointly 
between 1IT Kanpur and COL. DBR (Wang & Hannafin, 2005, p. 6) is "a systematic but flexible methodology 
aimed to improve educational practices through iterative analysis, design, development, and 
implementation, based on collaboration among researchers and practitioners in real-world settings, and 
leading to contextually sensitive design principles and theories.” 

A DBR approach would embed a research process into the course development and delivery processes that 
could be replicated over multiple iterations of the course. Using both quantitative and qualitative research 
strategies, the team could add design features, improve the learning resources, improve the delivery model 
and assessment strategies, and evaluate the overall effectiveness of the re-designed course to meet the 
stated learning outcomes. 
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The resulting design principles could become a model for development and delivery of large-scale courses 
to support development education outcomes. 

Recommendation 2 

Plan, re-design, and implement additional iterations of the M4D course, incorporating improvements 
suggested by students and development team members. Use an iterative DBR strategy as the guiding 
schema for course design and evaluation, and set specific goals for the course re-design and updated 
delivery processes. Evaluate and report the findings with a view to optimizing the course and defining a 
generalized set of design principles for courses of this type and intended audience. 

Recommendation 3 

Consider the primary outcomes for the course in the re-design process and whether the aim of the course 
is knowledge transmission (what students will know), competence (what students should be able to do), or 
a broader goal such as learner empowerment (the ability to learn on one’s own from multiple sources). Be 
explicit about designing for the primary outcome/s (Guardia, Maina & Sangra, 2013). 

Recommendation 4 

Use additional quizzes at the conclusion of each topic or week of lessons in M4D. The quizzes would 
provide ongoing feedback for students and potentially offer an incentive for them to continue their 
participation towards a certificate. 

Recommendation 5 

Re-organize online chat spaces and online discussion forums into moderated and structured activities 
associated with course topics or emergent topics. The discussion should be managed and curated by a 
designated online moderator who would also post discussion summaries on a weekly basis. Teacher 
presence in online spaces is a powerful design feature (Anderson, 2008). 

Recommendation 6 

Consider a course pre-test and post-test for students to help gauge the relative achievement of students 
entering the course with different levels of knowledge, measured against their acquired knowledge. 

Recommendation 7 

Use a cost-analysis framework to monitor course development, delivery and evaluation costs. Using a 
consistent schema, such as the one outlined by Bates (2001) to measure and control costs, would provide a 
benchmark for further planning and development. 

Recommendation 8 

Consider a credential pathway for M4D, with a view to formally associating the course for credit with 
institutions that are members of the OER Universitas Consortium f http: / / oerunversitas.org '). A credential 
is a powerful incentive for participation, and aligns with the M4D goals for students seeking to advance 
their standing or careers in the developing world. 
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